Image display apparatus and image display method

ABSTRACT

Provided is an image display apparatus, including: a display unit configured to allow a real space to be transparently viewed and configured to display a three-dimensional image; and a display control unit configured to display a plurality of the three-dimensional images on the display unit in a manner that the plurality of the three-dimensional images are arranged in a plurality of lines that are different from each other in position in a depth direction on a near side with respect to the real space.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of Japanese Priority PatentApplication JP 2014-021834 filed Feb. 7, 2014, the entire contents ofwhich are incorporated herein by reference.

BACKGROUND

The present disclosure relates to an image display apparatus and animage display method, and more specifically, to an image displayapparatus and an image display method that provide higher operability.

In recent years, development of transmissive HMDs (Head MountedDisplays) has been started. For example, there has been proposed an ideaof a transmissive HMD that expresses a sense of depth by adjustingbinocular parallax and a convergence angle (refer, for example, toJapanese Patent Application Laid-open No. 2012-42654). Morespecifically, there has been proposed an idea of displaying, by usingsuch a system, display objects such as menu icons under a state in whichthose display objects are superimposed on a scene in a real space.

SUMMARY

However, in recent years, functions of such transmissive HMDs havebecome more diversified, and in accordance therewith, a larger number ofdisplay objects to be displayed, such as menu icons, are displayed.Thus, when all the display objects to be displayed are displayed as theyare, there arise problems that the display objects are downsized, targeticons need to be searched for from among the large number of displayobjects, and the real space is difficult to view due to superimpositionof the large number of icons on a near side. Such problems may causedeterioration in operability.

There is a need to enhance operability of such display objects.

According to an embodiment of the present technology, there is providedan image display apparatus, including a display unit configured to allowa real space to be transparently viewed and configured to display athree-dimensional image, and a display control unit configured todisplay a plurality of the three-dimensional images on the display unitin a manner that the plurality of the three-dimensional images arearranged in a plurality of lines that are different from each other inposition in a depth direction on a near side with respect to the realspace.

The display control unit is capable of controlling sizes of theplurality of the three-dimensional images in accordance with thepositions in the depth direction of the plurality of lines in which theplurality of the three-dimensional images are arranged.

The display control unit is capable of controlling at least one ofvalue, chroma, and contrast of the plurality of the three-dimensionalimages in accordance with the positions in the depth direction of theplurality of lines in which the plurality of the three-dimensionalimages are arranged.

The display control unit is capable of controlling, in accordance withthe positions in the depth direction of the plurality of lines in whichthe plurality of the three-dimensional images are arranged, an intervalbetween the plurality of the three-dimensional images in each of theplurality of lines.

The display control unit is capable of controlling, in accordance withthe positions in the depth direction of the plurality of lines in whichthe plurality of the three-dimensional images are arranged, an intervalbetween the plurality of lines in the depth direction.

The plurality of lines in which the plurality of the three-dimensionalimages are arranged include a line on a depth side as viewed from auser, and a line on the near side as viewed from the user.

The plurality of the three-dimensional images include a plurality ofthree-dimensional images that are arranged in the line on the depth sideas viewed from the user, and a plurality of three-dimensional imagesthat are arranged in the line on the near side as viewed from the user.The display control unit is capable of controlling display positions ofthe plurality of the three-dimensional images so that at least parts ofthe plurality of three-dimensional images that are arranged in the lineon the depth side as viewed from the user are hidden by the plurality ofthree-dimensional images that are arranged in the line on the near sideas viewed from the user.

The display control unit is capable of controlling hues of the pluralityof the three-dimensional images in accordance with the positions in thedepth direction of the plurality of lines in which the plurality of thethree-dimensional images are arranged.

The display control unit is capable of controlling degrees of focusingof the plurality of the three-dimensional images in accordance with thepositions in the depth direction of the plurality of lines in which theplurality of the three-dimensional images are arranged.

The display control unit is capable of controlling, in accordance withroles assigned respectively to the plurality of the three-dimensionalimages, in which of the plurality of lines the plurality of thethree-dimensional images are arranged.

The image display apparatus according to the embodiment of the presenttechnology further includes a detection unit configured to detectchanges in at least one of position and orientation of the display unit.The display control unit is capable of causing, when the detection unitdetects the changes in the at least one of position and orientation ofthe display unit, the plurality of the three-dimensional images to bemoved in accordance with the changes.

The display control unit is capable of causing, when the detection unitstarts to detect the changes in the at least one of position andorientation of the display unit, display positions of the plurality ofthe three-dimensional images to be moved from positions with respect towhich the display unit has not yet been changed in any of position andorientation into a direction opposite to a direction in which thedisplay unit has been changed in the at least one of position andorientation. The display control unit is capable of causing, when thedetection unit detects that the changes in the at least one of positionand orientation have ended, the display positions of the plurality ofthe three-dimensional images to be returned to the positions withrespect to which the display unit has not yet been changed in the any ofposition and orientation.

The plurality of lines in which the plurality of the three-dimensionalimages are arranged include a line on a deeper side as viewed from auser, and a line on a nearer side as viewed from the user. The pluralityof the three-dimensional images include a plurality of three-dimensionalimages that are arranged in the line on the deeper side as viewed fromthe user, and a plurality of three-dimensional images that are arrangedin the line on the nearer side as viewed from the user. The displaycontrol unit is capable of causing the plurality of three-dimensionalimages that are arranged in the line on the nearer side as viewed fromthe user to be moved by an amount larger than an amount of moving theplurality of three-dimensional images that are arranged in the line onthe deeper side as viewed from the user.

The plurality of lines in which the plurality of the three-dimensionalimages are arranged include a line on a deeper side as viewed from auser, and a line on a nearer side as viewed from the user. The pluralityof the three-dimensional images include a plurality of three-dimensionalimages that are arranged in the line on the deeper side as viewed fromthe user, and a plurality of three-dimensional images that are arrangedin the line on the nearer side as viewed from the user. The displaycontrol unit is capable of causing the plurality of three-dimensionalimages that are arranged in the line on the nearer side as viewed fromthe user to be moved earlier than the plurality of three-dimensionalimages that are arranged in the line on the deeper side as viewed fromthe user.

According to another embodiment of the present technology, there isprovided an image display method, including displaying a plurality ofthree-dimensional images on a display unit in a manner that theplurality of three-dimensional images are arranged in a plurality oflines that are different from each other in position in a depthdirection on a near side with respect to a real space, the display unitbeing configured to allow the real space to be transparently viewed andconfigured to display the plurality of three-dimensional images.

According to the embodiments of the present technology, a plurality ofthree-dimensional images are displayed on a display unit in a mannerthat the plurality of three-dimensional images are arranged in aplurality of lines that are different from each other in position in adepth direction on a near side with respect to a real space, the displayunit being configured to allow the real space to be transparently viewedand configured to display the plurality of three-dimensional images.

According to the embodiments of the present technology, photographicsubjects can be imaged. Further, according to the embodiments of thepresent technology, operability can be enhanced.

These and other objects, features and advantages of the presentdisclosure will become more apparent in light of the following detaileddescription of best mode embodiments thereof, as illustrated in theaccompanying drawings.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a view of an example of virtual image display positions in athree-dimensional image;

FIGS. 2A-2C are views of examples of an external appearance of atransmissive HMD;

FIG. 3 is a block diagram of a main configuration example of thetransmissive HMD;

FIG. 4 is a view of a display example of display objects;

FIG. 5 is a view of an example of a hierarchical arrangement of thedisplay objects;

FIG. 6 is a flowchart showing an example of a flow of a display controlprocess;

FIGS. 7A-7C are explanatory views of binocular parallax and aconvergence angle;

FIG. 8 is a view of another display example of the display objects;

FIG. 9 is a view of still another display example of the displayobjects;

FIG. 10 is a view of yet another display example of the display objects;

FIG. 11 is a view of yet another display example of the display objects;

FIG. 12 is a view of yet another display example of the display objects;

FIGS. 13A-13C are views of an example of movements in a horizontaldirection of the display objects;

FIGS. 14A-14C are views of an example of movements in a verticaldirection of the display objects;

FIGS. 15A and 15B are views of examples of parameters;

FIG. 16 is a flowchart showing an example of a flow of another displaycontrol process;

FIGS. 17A-17C are views of an example of movements in turning directionsof the display objects;

FIGS. 18A and 18B are views of an example of a parameter; and

FIG. 19 is a flowchart showing an example of a flow of still anotherdisplay control process.

DETAILED DESCRIPTION OF EMBODIMENTS

In the following, description is made of embodiments for carrying outthe present disclosure (hereinafter, simply referred to as embodiments).Note that, the description is made in the following order.

1. First embodiment (image display apparatus)

2. Second embodiment (transmissive HMD)

(1. First Embodiment)

(Display of Three-Dimensional Image)

In recent years, development of transmissive HMDs (Head MountedDisplays) has been started. The transmissive HMDs use a display unit(transmissive display) configured to transmit light from a rear surfaceside thereof and allow a real space to be viewed on the rear surfaceside. With this, a user of the transmissive HMD can see images displayedon the transmissive display under a state in which the images aresuperimposed on a scene in the real space on the rear surface side ofthe transmissive display.

There has been proposed an idea of displaying, on such a transmissivedisplay, predetermined images (display objects) in GUIs (Graphical UserInterfaces), such as menu icons to be displayed on the transmissive HMD.However, even by such a display configuration, when a sense of depth ofthe display objects is poor, the images may become visually unnatural tocause a sense of discomfort to the user. As a result, operability may bedeteriorated.

In this context, as a method of displaying images on the transmissivedisplay, Japanese Patent Application Laid-open No. 2012-42654 disclosesa method of expressing a sense of depth by adjusting binocular parallaxand convergence angles of the images. However, also in a case where sucha display configuration is applied to the display objects, a sense ofdepth of the display objects is insufficient with respect to the scenein the real space. Also in this case, the images may become visuallyunnatural to cause a sense of discomfort to the user. As a result, theoperability may be deteriorated.

For example, as illustrated in FIG. 1, cylinders 11 to 13 are virtualimages superimposed on a scene 1 in the real space. Positions at whichvirtual presence of those cylinders is recognized (also referred to asvirtual image positions) are set to positions that are different fromeach other in a depth direction. Nevertheless, the sense of depth ispoor, and hence does not match with a sense of perspective in the scenein the real space. As a result, the images are visually unnaturallydisplayed to cause a sense of discomfort to the user.

Further, in recent years, functions of such transmissive HMDs havebecome more diversified, and in accordance therewith, a larger number ofdisplay objects such as menu icons need to be displayed. Thus, when allthe display objects to be displayed are displayed as they are, therearise problems that the display objects are downsized, target icons needto be searched for from among the large number of display objects, andthe real space is difficult to view due to superimposition of the largenumber of icons on a near side. Such problems may cause deterioration inoperability.

(Image Display Apparatus)

In view of the circumstances, there is provided an image displayapparatus including a display unit configured to allow a real space tobe transparently viewed, and configured to display a three-dimensionalimage, and a display control unit configured to display a plurality ofthe three-dimensional images on the display unit in a manner that theplurality of the three-dimensional images are arranged in a plurality oflines that are different from each other in position in a depthdirection on a near side with respect to the real space.

In other words, the image display apparatus causes the display unit,which is configured to allow the real space to be transparently viewedand configured to display the three-dimensional image, to display theplurality of the three-dimensional images in a manner that the pluralityof the three-dimensional images are arranged in the plurality of linesthat are different from each other in position in the depth direction onthe near side with respect to the real space.

Specifically, the display objects such as menu icons are displayed on adisplay unit 112 as the three-dimensional images in the plurality oflines that are different from each other in position in the depthdirection. With this, the number of the display objects in each of thelines is reduced. Thus, for example, the display objects can bedisplayed on an enlarged scale, and the user can select display objectsof target menu icons from among a smaller number of the display objects(display objects in a single line). In this way, operability can beenhanced.

Further, the display control unit may control sizes of the displayobjects as the three-dimensional images in accordance with the positionsin the depth direction of the lines in which the display objects arearranged. With this, the sense of depth of the display objects can beincreased, and the operability can be enhanced.

Still Further, the display control unit may control at least one ofvalue, chroma, and contrast of the display objects in accordance withthe positions in the depth direction of the lines in which the displayobjects are arranged. With this, the sense of depth of the displayobjects can be increased, and the operability can be enhanced.

Yet further, the display control unit may control, in accordance withthe positions in the depth direction of the lines in which the displayobjects are arranged, an interval between the display objects in each ofthe lines. With this, the sense of depth of the display objects can beincreased, and the operability can be enhanced.

Yet further, the display control unit may control, in accordance withthe positions in the depth direction of the lines in which the displayobjects are arranged, an interval between the lines in the depthdirection. With this, the sense of depth of the display objects can beincreased, and the operability can be enhanced.

Yet further, the display control unit may control display positions ofthe display objects so that at least parts of the display objects thatare arranged in the line on a depth side as viewed from the user arehidden by the display objects that are arranged in the line on the nearside as viewed from the user. With this, the sense of depth of thedisplay objects can be increased, and the operability can be enhanced.

Yet further, the display control unit may control hues of the displayobjects in accordance with the positions in the depth direction of thelines in which the display objects are arranged. With this, the sense ofdepth of the display objects can be increased, and the operability canbe enhanced.

Yet further, the display control unit may control degrees of focusing ofthe display objects in accordance with the positions in the depthdirection of the lines in which the display objects are arranged. Withthis, the sense of depth of the display objects can be increased, andthe operability can be enhanced.

Yet further, the display control unit may control, in accordance withroles assigned respectively to the display objects, in which of thelines the display objects are arranged. With this, based on the displaypositions of the display objects (positions in the depth direction), theuser can easily recognize the roles assigned respectively to the displayobjects, such as types of menu icons. In other words, the user can moreeasily find target display objects. In this way, the operability can beenhanced.

Yet further, the image display apparatus may further includes adetection unit configured to detect changes in at least one of positionand orientation of the display unit so that, when the detection unitdetects the changes in the at least one of position and orientation ofthe display unit, the display control unit causes the display objects tobe moved in accordance with the changes. With this, the display objectsand physical objects in the real space can be easily distinguished fromeach other, and the operability can be enhanced.

Yet further, when the detection unit starts to detect the changes in theat least one of position and orientation of the display unit, thedisplay control unit may cause the display positions of the displayobjects to be moved from positions with respect to which the displayunit has not yet been changed in any of position and orientation into adirection opposite to a direction in which the display unit has beenchanged in the at least one of position and orientation. When thedetection unit detects that the changes in the at least one of positionand orientation have ended, the display control unit may cause thedisplay positions of the display objects to be returned to the positionswith respect to which the display unit has not yet been changed in theany of position and orientation. With this, the display objects andphysical objects in the real space can be easily distinguished from eachother, and the operability can be enhanced.

Yet further, the display control unit may cause the display objects thatare arranged in the line on a nearer side as viewed from the user to bemoved by an amount larger than an amount of moving the display objectsthat are arranged in the line on a deeper side as viewed from the user.With this, the sense of depth of the display objects can be increased,and the operability can be enhanced.

Yet further, the display control unit may cause the display objects thatare arranged in the line on the nearer side as viewed from the user tobe moved earlier than the display objects that are arranged in the lineon the deeper side as viewed from the user. With this, the sense ofdepth of the display objects can be increased, and the operability canbe enhanced.

Yet further, the display control unit may execute arbitrary ones of thevarious processes described above in various combinations.

Note that, the above-mentioned display objects may be displayed on anon-transmissive display unit that is configured not to transmit light.Specifically, on the non-transmissive display unit, images to be viewedfrom a plurality of viewpoints (for example, display objects such as amenu icon) may be displayed as a three-dimensional image in asuperimposed manner on captured images obtained by an imaging unit (alsoreferred to as through images). Also in that case, as in the case wherethe display objects are displayed on the transmissive display unitdescribed above, it is only necessary to generate left-eye images andright-eye images of the display objects and display those images on thedisplay unit in a manner that the display objects are arranged in aplurality of lines that are different from each other in position in thedepth direction on the near side with respect to a scene in the realspace.

In other words, the above description “to allow the real space to betransparently viewed” conceptually includes not only “to allow a scenein the real space on the rear surface side of the display to be viewedthrough the display” as in the case of the transmissive display, butalso “to allow the through images to be viewed, which are obtained bythe imaging unit and displayed on the display” as in the case of theabove-mentioned non-transmissive display.

(2. Second Embodiment)

(External Appearance of Transmissive HMD)

The present technology is applicable not only to the image displayapparatus, but also, for example, to transmissive HMDs. FIG. 2A-2C areviews of an embodiment of the image display apparatus, specifically,illustrating examples of an external appearance of the transmissive HMDto which the present technology is applied. More specifically, asillustrated in FIG. 2A, a frame 111 of a transmissive HMD 100 has whatis called an eyeglass shape so as to be used, as well as eyeglasses,while being fitted to the face of a user by hooking end portions of theframe 111 to the ears of the user.

Parts corresponding to lenses of eyeglasses serve as the display unit112 (right-eye display unit 112A and left-eye display unit 112B). Whenthe user wears the transmissive HMD 100, the right-eye display unit 112Acomes close to the front of the right eye of the user, and the left-eyedisplay unit 112B comes close to the front of the left eye of the user.

The display unit 112 is a transmissive display configured to transmitlight. Thus, a scene in a real space on a rear surface side of theright-eye display unit 112A, that is, in front of the right-eye displayunit 112A (transparent video) is visible to the right eye of the userthrough the right-eye display unit 112A. Similarly, a scene in the realspace on a rear surface side of the left-eye display unit 112B, that is,in front of the left-eye display unit 112B (transparent video) isvisible to the left eye of the user through the left-eye display unit112B. Thus, images that are displayed on the display unit 112 arevisible to the user under a state in which the images are superimposedon a near side of the scene in the real space in front of the displayunit 112.

The right-eye display unit 112A is configured to display images to theright eye of the user (right-eye images), and the left-eye display unit112B is configured to display images to the left eye of the user(left-eye images). In other words, the display unit 112 is capable ofdisplaying different images respectively on the right-eye display unit112A and the left-eye display unit 112B, for example, displaying athree-dimensional image.

The three-dimensional image is formed of the right-eye image and theleft-eye image that have binocular parallax and a convergence angletherebetween. By controlling the binocular parallax and the convergenceangle, the three-dimensional image is visible to be located far from ornear the user. In other words, the three-dimensional image is an imagethat can be controlled in depth position (position that is not an actualposition but a virtual position at which the three-dimensional image isvisible to exist thereat exactly as it is to the user (also referred toas virtual-image position)).

In other words, the display unit 112 is capable of displaying images(three-dimensional images) so that the images are visible to the user ina manner of being located in the real space in front of the display unit112.

Further, as illustrated in FIG. 2A, a hole 113 is formed near thedisplay unit 112 in the frame 111. An imaging unit configured to imagephotographic subjects is provided near the hole 113 in an inside of theframe 111. The imaging unit images the photographic subjects in the realspace in front of the transmissive HMD 100 through the hole 113 (frontof the transmissive HMD 100 with respect to the user wearing thetransmissive HMD 100). More specifically, the imaging unit images thephotographic subjects in the real space located in a display area of thedisplay unit 112 (right-eye display unit 112A and left-eye display unit112B) as viewed from the user. With this, image data of captured imagesis generated. The generated image data is, for example, stored inpredetermined storage media or transmitted to other devices.

Note that, a position of the hole 113 (that is, imaging unit) may bearbitrarily set, specifically, the hole 113 may be formed at a positionother than that in the example illustrated in FIG. 2A. Further, the hole113 (that is, imaging unit) is not particularly limited in number,specifically, the single hole 113 may be formed as illustrated in FIG.2A, or a plurality of holes 113 may be formed.

Further, as long as the transmissive HMD 100 can be fitted to the face(head) of the user in a manner that the right-eye display unit 112Acomes close to the front of the right eye of the user and that theleft-eye display unit 112B comes close to the front of the left eye ofthe user, the frame 111 is not particularly limited in shape. Forexample, the transmissive HMD 100 has the shape as illustrated in FIG.2B.

In the case of the example of FIG. 2B, a frame 131 of the transmissiveHMD 100 is formed into a shape of sandwiching and fixing the head of theuser from behind. Similar to the display unit 112, a display unit 132 inthis case is also a transmissive display. Specifically, the display unit132 similarly includes a right-eye display unit 132A and a left-eyedisplay unit 132B. When the user wears the transmissive HMD 100, theright-eye display unit 132A comes close to the front of the right eye ofthe user, and the left-eye display unit 132B comes close to the left eyeof the user.

In addition, the right-eye display unit 132A is a display unitequivalent to the right-eye display unit 112A, and the left-eye displayunit 132B is a display unit equivalent to the left-eye display unit112B. In other words, similar to the display unit 112, the display unit132 is also capable of displaying three-dimensional images.

Further, also in the case of FIG. 2B, as in the case of FIG. 2A, a hole133 equivalent to the hole 113 is formed near the display unit 132 onthe frame 131. An imaging unit configured to image photographic subjectsis provided near the hole 133 in an inside of the frame 131. As in thecase of FIG. 2A, the imaging unit images the photographic subjects inthe real space in front of the transmissive HMD 100 through the hole 133(front of the transmissive HMD 100 with respect to the user wearing thetransmissive HMD 100).

As a matter of course, a position of the hole 133 (that is, imagingunit) may be arbitrarily set as in the case of FIG. 2A, specifically,the hole 133 may be formed at a position other than that in the exampleillustrated in FIG. 2B. Further, the hole 133 (that is, imaging unit) isalso not particularly limited in number as in the case of FIG. 2A.

Alternatively, as in the example illustrated in FIG. 2C, a part of aconfiguration of the transmissive HMD in the example of FIG. 2A may beprovided separately from the frame 111. In the example of FIG. 2C, theframe 111 is connected to a control box 152 via a cable 151.

The cable 151 is a communication path for predetermined wiredcommunication, which electrically connects a circuit in the frame 111and a circuit in the control box 152 to each other. The control box 152includes a part of a configuration (such as circuit) in the frame 111 inthe case of the example of FIG. 2A. Specifically, the control box 152may include a control unit and a storage unit configured to store imagedata. The circuit in the frame 111 and the circuit in the control box152 may communicate to each other. The imaging unit in the frame 111 mayperform imaging under control by the control unit in the control box152. The image data of images captured by the imaging may be supplied tothe control box 152, and stored to the storage unit therein.

The control box 152 can be stored, for example, in a pocket of a garmentof the user. With this configuration, the frame 111 of the transmissiveHMD 100 can be downsized in comparison with that in the case of FIG. 2A.

Note that, the communication between the circuit in the frame 111 andthe circuit in the control box 152 may be performed in a wired manner ormay be performed in a wireless manner. In a case of the wirelesscommunication, the cable 151 can be omitted.

(Example of Internal Configuration)

FIG. 3 is a block diagram of an example of an internal configuration ofthe transmissive HMD 100. As illustrated in FIG. 3, the transmissive HMD100 includes a system controller 211.

The system controller 211 is formed of a microcomputer including a CPU(Central Processing Unit), a ROM (Read Only Memory), a RAM (RandomAccess Memory), a nonvolatile memory unit, and an interface unit, and isconfigured to control units of the transmissive HMD 100. The systemcontroller 211 controls the units based on internal operating programs.For example, by controlling the units, the system controller 211executes display control of images to be displayed on the display unit112.

Further, the transmissive HMD 100 includes a sensor unit 221, a sensorcontrol unit 222, and a sensor signal processing unit 223.

The sensor unit 221 includes arbitrary sensors mounted near the displayunit 112, such as an acceleration sensor, a gyro sensor, a magneticsensor, and a barometric sensor. With this, for example, movements ofthe head and the neck of the user and movements of the transmissive HMD100 are detected as signals in response to movements of the user.Further, the sensor unit 221 may also include a capacitive sensor, abutton, and a GPS (Global Positioning System) so that signals from asystem of those sensors to be used by the user at the time of operatingthe transmissive HMD 100 are also processed in this unit.

In other words, the sensor unit 221 includes an arbitrary input device,and accepts information to be input via the input device. A position ofsuch an input device is not limited to near the display unit 112, andmay be arbitrarily set.

Further, the sensor unit 221 may also include a sensor configured todetect, for example, sight lines of the user. Examples of such a sensorof the sensor unit 221 include an imaging unit (sight-line imaging unit)arranged near the display unit 112, which is configured to image eyes ofthe user. In this case, for example, the system controller 211 mayperform image analysis of images of the eyes of the user, which arecaptured by the sight-line imaging unit, so that a sight-line direction,a focal length, pupil opening degrees, fundus patterns, and whether theeyelids are opened or closed are detected. Alternatively, the sensorunit 221 may include, instead of such a sight-line imaging unit, a lightemitting unit arranged near the display unit 112, which is configured toradiate light to the eyes of the user, and a light receiving unitconfigured to receive reflected light from the eyes, performphotoelectric conversion of the reflected light, and output anelectrical signal (light receiving signal) that is generated by thephotoelectric conversion. In that case, the system controller 211 may beconfigured to detect, for example, the sight-line direction based on thelight receiving signal obtained by the light receiving unit.Alternatively, for example, the system controller 211 may be configuredto detect thicknesses of crystalline lens of the user based on the lightreceiving signal obtained as the electrical signal through theconversion of the reflected light that is received by the lightreceiving unit. In other words, the sensor unit 221 may include asight-line detection unit configured to detect the sight lines of theuser. Note that, how the sight lines are detected is not particularlylimited.

The sensor control unit 222 is configured to control the sensors inresponse to instructions from the system controller 211, specifically,control at which timing which of the sensors is driven, by what kind ofdriving method the detection is performed, and the like. Further, thesensor signal processing unit 223 is configured to execute variousnumerical computations such as mean and variance as pre-processes on thesensor signals that are detected by the sensor unit 221 prior totransmission to the system controller 211 side.

The transmissive HMD 100 may need operations by the user so as, forexample, to perform power-on/off, start or end display of variousinformation images, change image contents, perform display control suchas luminance level control and color tone control, and change displayarea of a display screen. Specifically, for receiving those operations(detecting triggers for processing operations), the sensor unit 221 mayinclude a controller such as operation keys and an operation dial to beoperated by the user, and an operation detection mechanism configured todetect operations to the controller so that the sensor unit 221 candetect those operations by the user.

Instead of provision of such a controller, the system controller 211 maybe configured to determine operating intention of the user orappropriate operation processes based on conditions of the user (such asmovements of the eyes and behavior or conditions of the body), which aredetected by the sensors of the sensor unit 221, and to execute theprocesses in accordance therewith.

Further, the sensor unit 221 may be configured to detect other externalinformation items (such as detected information items of environmentalconditions of the transmissive HMD 100, locations, dates, and conditionsof a photographic subject). The system controller 211 may be configuredto determine appropriate operation processes based on the externalinformation items detected by the sensor unit 221, and to execute thoseprocesses.

Further, the transmissive HMD 100 includes an image generating unit 231,a display-image processing unit 232, a display drive unit 233, and adisplay control unit 234.

The image generating unit 231 is configured to generate image signalsunder control by the system controller 211. The system controller 211causes the image generating unit 231 to generate the image signals to beconverted to images that are presented to the user in accordance withcontents or numerical values obtained from the units. In this way,picture images, graph images, letter images, or images of warning to theuser are generated.

The display-image processing unit 232 includes what is called a videoprocessor, and is configured to execute various display processes on thesupplied image signals. For example, the display-image processing unit232 performs luminance level adjustment, color correction, contrastadjustment, sharpness (edge enhancement) adjustment of image pickupsignals supplied from an image pickup signal processing unit 252.Further, the display-image processing unit 232 may make settings ofdisplay positions on the display. Still further, the display-imageprocessing unit 232 may perform, for example, generation of an enlargedimage of a part of the image pickup signals, or generation of adownsized image thereof, soft focus, mosaicing, luminance inversion,highlighting (emphatic display) of a part of the images, image effectprocesses such as variation of an atmosphere of the entire color, imageseparation or synthesis for split display of captured images, a processof generating character images and picture images, and a process ofsynthesizing the generated images into the captured images. Thedisplay-image processing unit 232 supplies the image signals obtainedthrough those signal processes to the display drive unit 233.

Note that, the display-image processing unit 232 may be configured toexecute such processes in response to instructions issued by the displaycontrol unit 234, or to the sensor signals supplied from the sensorsignal processing unit 223. Further, the display-image processing unit232 may be configured to execute the same processes on images suppliedfrom the display control unit 234. Specifically, the display-imageprocessing unit 232 may be configured to execute the signal processes,which are necessary for displaying, also on the image signals generatedin the image generating unit 231 and supplied to the display-imageprocessing unit 232, and then supply those image signals to the displaydrive unit 233.

The display drive unit 233 includes a pixel drive circuit configured todisplay the image signals supplied from the display-image processingunit 232 on the display unit 112. Specifically, the display drive unit233 is configured to perform display by applying drive signals to pixelsarranged in matrix in the display unit 112 at predeterminedhorizontal/vertical drive timings based on video signals. Further, thedisplay drive unit 233 may be configured to display a part of or theentire screen in a through mode by controlling luminances of the pixelsof the display unit 112.

The display control unit 234 is configured to control, in response tothe instructions from the system controller 211, the processingoperations in the display-image processing unit 232, operations in thedisplay drive unit, and images to be displayed on the left and rightdisplay units, and to instruct the display-image processing unit 232 toexecute the signal processes. Further, the display control unit 234 isconfigured also to cause the display drive unit 233 to switch thethrough mode, an image display mode, and monocular display mode to eachother.

Further, the transmissive HMD 100 includes a sound generating unit 241,an audio output unit 242, an audio input unit 243, and a sound signalprocessing unit 244.

The sound generating unit 241 is configured, for example, to generatesound signals such as a voice message, and to generate sound signalssuch as electronic sound to be presented to the user by executing avoice synthesis process in response to the instructions from the systemcontroller 211.

The audio output unit 242 includes speakers or earphone speakers mountedto the transmissive HMD 100, and an amplifier circuit for thosespeakers. The sound signals generated by the sound generating unit 241are supplied to the audio output unit 242. With this, the voice message,the electronic sound, and the like are audible to the user. Note that,examples of the audio output unit 242 may include what is called abone-conduction speaker.

The audio input unit 243 includes a microphone amplifier unit configuredto execute an amplification process on sound signals obtained via amicrophone, and an A/D converter, and is configured to output audiodata.

The audio signal processing unit 244 includes a digital signal processorand a D/A converter. The audio data obtained via the audio input unit243 and the audio data generated by the sound generating unit 241 aresupplied to the audio signal processing unit 244. The audio signalprocessing unit 244 is configured to execute processes such as volumecontrol, tone control, and sound effects on the supplied audio dataunder the control by the system controller 211. Then, the processedaudio data is converted to an analog signal by the audio signalprocessing unit 244, and supplied to the audio output unit 242. Notethat, the audio signal processing unit 244 needs not necessarily beconfigured to execute the digital signal process, and may be configuredto execute signal processes through an analog amplifier or an analogfilter.

The sound signals output from the audio signal processing unit 244 areoutput as sound from the earphone speakers of the audio output unit 242.With such a configuration, external sound collected by the audio inputunit 243, or the sound generated by the sound generating unit 241 areaudible to the user.

Further, the transmissive HMD 100 includes an imaging unit 251, theimage pickup signal processing unit 252, and an imaging control unit253.

The imaging unit 251 includes a lens system including an imaging lens,an aperture, a zoom lens, a focus lens, and the like. The imaging unit251 also includes a drive system configured to cause the lens system toperform a focusing operation or a zooming operation. The imaging unit251 also includes a solid-state image pickup element array configured todetect image pickup light obtained through the lens system, and togenerate image pickup signals through photoelectric conversion. Examplesof the solid-state image pickup element array include a CCD (ChargeCoupled Device) sensor array and a CMOS (Complementary Metal OxideSemiconductor) sensor array.

The imaging unit 251 is configured to image a scene in front of the user(photographic subject in the real space in front of the user) through,for example, the hole 113. As a matter of course, the imaging unit 251may be configured also to image scenes in other directions, such as ascene behind the user.

The image pickup signal processing unit 252 includes asample-and-hold/AGC (Automatic Gain Control) circuit configured toperform gain control or wave pattern reshaping on the signals obtainedby the solid-state image pickup element of the imaging unit 251, and avideo A/D converter. With this, the image pickup signals as digital dataare obtained. Further, the image pickup signal processing unit 252 maybe configured also to execute a white-balance process, a luminanceprocess, a color signal process, a blur correcting process, and the likeon the image pickup signals.

The imaging control unit 253 is configured to control operations of theimaging unit 251 and the image pickup signal processing unit 252 inresponse to the instructions from the system controller 211.Specifically, the imaging control unit 253 controls on/off of theoperations of the imaging unit 251 and the image pickup signalprocessing unit 252. Further, the imaging control unit 253 performscontrol (motor control) for causing the imaging unit 251 to performoperations such as automatic focusing, automatic exposure adjustment,aperture adjustment, zooming in/out, and focus change.

Note that, in a case where a movable mechanism capable of changing aphotographic subject direction with respect to an imaging lens isprovided, the imaging control unit 253 varies a direction of the imaginglens of the imaging unit 251 by controlling operation of the movablemechanism in response to the instructions from the system controller211.

Further, the imaging control unit 253 includes a timing generatorconfigured to generate timing signals. In response to those timingsignals, signal processing operations of the solid-state image pickupelement of the imaging unit 251, and the sample-and-hold/AGC circuit andthe video A/D converter of the image pickup signal processing unit 252are performed. Further, this timing control also enables variablecontrol of an imaging frame rate.

Further, the imaging control unit 253 may be configured also to controlimaging sensitivities and signal processes in the solid-state imagepickup element of the imaging unit 251 and the image pickup signalprocessing unit 252. Specifically, the imaging control unit 253 mayperform, as imaging sensitivity control, gain control of signals to beread out from the solid-state image pickup element of the imaging unit251, black level setting control, control of various coefficients inprocesses on the image pickup signals in a form of digital data, controlof a correction amount in the blur correcting process, and the like.

Still further, the imaging control unit 253 may be configured also toperform control such as overall sensitivity adjustment irrespective ofwavelength bands, and adjustment of imaging sensitivities in particularwavelength bands such as an infrared region and an ultraviolet region(specifically, imaging in which particular wavelength bands areexcluded). More specifically, the sensitivity adjustment in accordancewith the wavelengths can be performed by insertion of wavelength filtersin the imaging lens system and wavelength filter calculation processeson the image pickup signals. In those cases, the imaging control unit253 is capable of performing sensitivity control, for example, bycontrolling the insertion of the wavelength filters, or by specifyingfilter calculation coefficient.

For example, captured image signals obtained by the imaging unit 251 andthe image pickup signal processing unit 252 are supplied to thedisplay-image processing unit 232 together with the information imagesignals generated in the image generating unit 231. The display-imageprocessing unit 232 executes not only the above-mentioned various signalprocesses on the image signals, but also an image-split signal process(image synthesis process) on the image signals of those two types sothat images are displayed at once on the display unit 112.

The image signals that have been subjected to the synthesis process inthe display-image processing unit 232 are supplied to the display driveunit 233, and then displayed on the display unit 112. With this, on thedisplay unit 112, the captured images and images of other types aredisplayed at once. In other words, other various images are visible tothe user who is watching the captured images.

In order to start or end an image pickup operation, perform the zoomingoperation and the focusing operation, or perform adjustment of thecaptured image, and the like, operations by the user may be needed. As amatter of course, the operations by the user may be needed, for example,also to perform power-on/off, start or end display of variousinformation images, change image contents, perform display control suchas luminance level control and color tone control, and change displayareas on a display screen. Specifically, for receiving those operations(detecting triggers of the operations), the sensor unit 221 may includea controller such as operation keys. Alternatively, the systemcontroller 211 may be configured to determine operating intention of theuser or appropriate operation processes based on the conditions of theuser (such as movements of the eyes and behavior or conditions of thebody), which are detected by the various sensors of the sensor unit 221,and to execute the processes in accordance therewith.

Further, the sensor unit 221 may be configured to be capable ofdetecting external information items (detected information items such asenvironmental conditions of the transmissive HMD 100, locations, dates,and conditions of a photographic subject). The system controller 211 maybe configured to determine appropriate operation processes based on theexternal information items, and to execute those processes.

Further, the transmissive HMD 100 includes a storage unit 261, acommunication unit 262, and a drive 263.

The storage unit 261 includes arbitrary storage media such assolid-state memories including a HDD (Hard Disk Drive) and a flashmemory, a memory card incorporating the solid-state memories, an opticaldisk, an a magneto-optical disk, and a holographic memory, and isconfigured to a record/reproduce data that is stored in those storagemedia.

Specifically, the data of images captured by the imaging unit 251 andprocessed into the image pickup signals by the image pickup signalprocessing unit 252, image data received via the communication unit 262,and the various information image signals generated by the imagegenerating unit 231 may be stored to the storage unit 261. Further, theaudio data obtained via the audio input unit 243, audio data receivedvia the communication unit 262, and the audio data generated in thesound generating unit 241 also may be stored to the storage unit 261.

The storage unit 261 executes an encoding process on the supplied imagedata and the supplied audio data under the control by the systemcontroller 211 so that those data items can be stored to the storagemedia, and then stores those encoded data items (encoded data items) tothe storage media. Further, under the control by the system controller211, the storage unit 261 reproduces the image data and the audio datafrom the storage media, and outputs those data items to other processingunits.

Examples of the data to be reproduced in the storage unit 261 includeevery kind of data items to be displayed, specifically, moving-imagecontent items such as a movie and a video clip, still-image contentitems captured by a digital still camera and the like and recorded tothe recording media, data of electronic books and the like, image datagenerated using a personal computer and the like by the user, text data,computer use data such as spreadsheet data, and game images.

The communication unit 262 is configured to exchange data with respectto apparatus that are external to the transmissive HMD 100 (alsoreferred to as external apparatus). Examples of the external apparatusinclude a camcorder, an imaging apparatus such as a digital stillcamera, a computer apparatus, a smartphone, a smartwatch, AV apparatussuch as a video storage apparatus and a television set, and a networkserver apparatus each having a communication function.

Further, network communication may be performed, for example, via ashort-distance wireless communication between network access points witha predetermined system such as a wireless LAN (Local Area Network) andBluetooth (trademark). Alternatively, direct wireless communication maybe performed between external apparatus having a common communicationfunction.

In a case where the external apparatus is an imaging apparatus, data ofimages captured by the imaging apparatus may be sent from the externalapparatus to the transmissive HMD 100. In a case where the externalapparatus is a content source apparatus, every kind of data items to bedisplayed, specifically, moving-image content items such as a movie anda video clip, still-image content items captured by a digital stillcamera and the like and recorded to the recording media, data ofelectronic books and the like, image data generated using a personalcomputer and the like by the user, text data, computer use data such asspreadsheet data, and game images may be sent from the externalapparatus to the transmissive HMD 100.

Further, the audio data obtained via the audio input unit 243, audiodata reproduced in the storage unit, the audio data received via thecommunication unit 262 are supplied to the audio signal processing unit244 in response to the instructions from the system controller 211.

Thus, while wearing the device, the user can view the captured imagesand listen to the external sound that is recorded at the time ofcapturing, view the images and listen to the sound that are reproducedin the storage unit, and view the images and listen to the sound thatare received via the communication unit 262.

In particular, the images generated in the image generating unit 231 aresupplied to the display-image processing unit 232 together with thecaptured images, the reproduced images, or the received images. Thus,the various information images are displayed together with the capturedimages, the reproduced images, or the received images.

Further, when the audio data is generated in the sound generating unit241 and supplied at the same timing to the audio signal processing unit244, the voice message and alarm generated in the sound generating unit241 are audible under a state in which the external sound, thereproduced sound, the received sound, and the like are reproduced.

The system controller 211 needs to determine not only operations of thedisplay system and the imaging function but also triggers of controllingoperations of play, cue, fast-forward reproduction/fast-reversereproduction, pause, and recording in the storage unit 261, andcontrolling operations of data exchange in the communication unit 262.Also in this case, the controller such as operation keys to be operatedby the user may be provided, for example, to the sensor unit 221 so thatprocesses in response to the operations may be executed. Further, thesystem controller 211 may be configured to determine operating intentionof the user or appropriate operation processes from the conditions ofthe user (such as movements of the eyes and behavior or conditions ofthe body), which are detected by the sensor unit 221, and to execute theprocesses in accordance therewith.

Still further, the sensor unit 221 may be configured to be capable ofdetecting external information items of the transmissive HMD 100(detected information items such as environmental conditions of thedisplay apparatus, locations, dates, and conditions of a photographicsubject). The system controller 211 may be configured to determineappropriate operation processes based on the external information items,and to execute those processes.

Removable media 264 such as an optical disk and a semiconductor memoryare mounted as appropriate to the drive 263. Computer programs and datathat are read out from the removable media 264 via the drive 263 aresupplied to the system controller 211, and then stored or installed tothe storage unit 261.

In the transmissive HMD 100 configured as described above, the displayunit 112 transmits, as described above, the light from the rear surfaceside of the display unit 112 (front of the display unit 112 as viewedfrom the user), and displays images to be viewed from a plurality ofviewpoints as a three-dimensional image (display object) in asuperimposed manner on a scene in the real space on the rear surfaceside thereof.

Further, the image generating unit 231 generates the left-eye image andthe right-eye image as the three-dimensional image (display object) tobe displayed on the display unit 112.

Further, the display control unit 234 causes a plurality of thethree-dimensional images (display objects) generated by the imagegenerating unit 231 to be displayed on the display unit 112 in a mannerthat the plurality of three-dimensional images are arranged in aplurality of lines that are different from each other in position in thedepth direction on the near side with respect to the scene in the realspace.

Specifically, the display control unit 234 sets virtual-image positionsof the display objects to desired positions on the near side withrespect to the scene in the real space as viewed from the user. In otherwords, the display control unit 234 sets those virtual-image positionsof the display objects to form the plurality of lines that are differentfrom each other in position in the depth direction on the near side withrespect to the scene in the real space.

Next, the display control unit 234 controls the display-image processingunit 232 so as to reflect (apply) the settings to the display objectsgenerated by the image generating unit 231. In other words, thedisplay-image processing unit 232 assigns the virtual-image positionsset by the display control unit 234 to the display objects.

Then, the display control unit 234 controls the display unit 112 throughintermediation of the display drive unit 233 so as to display thedisplay objects to which the virtual-image positions are assigned by thedisplay-image processing unit 232.

In order to set such virtual-image positions of the display objects, thedisplay control unit 234 may be configured to set binocular parallax anda convergence angle of each of the display objects. In that case, amethod of adjusting (setting) the binocular parallax and the convergenceangles is not particularly limited. For example, the three-dimensionaldisplay apparatus disclosed in Japanese Patent Application Laid-open No.H08-322004 includes a system configured to shift a video electricallydisplayed in a display plane in a horizontal direction so that aconvergence angle substantially matches a diopter scale in real time.Further, the three-dimensional-video reproducing apparatus disclosed inJapanese Patent Application Laid-open No. H08-211332 is configured toobtain a three-dimensional video by using binocular parallax, andincludes a convergence angle selection section configured to set aconvergence angle at the time when a reproduced video is watched, and acontrol section configured to control relative reproduction positions ofthe left and right videos based on information of a selected convergenceangle. For example, the display control unit 234 may be configured toset parallax and the convergence angle of each of the display objects byusing the methods according to those technologies.

Note that, in order to emphasize a sense of depth of the displayobjects, a predetermined display control process may be executed inaccordance with their virtual-image positions (positions in the depthdirection). Specifically, in accordance with the virtual-image positions(positions in the depth direction) of the display objects, the displayobjects may be changed in size, value, chroma, and contrast. An intervalbetween the display objects may be changed in each line. Superimpositionof the display objects aligned near and far in the depth direction maybe expressed. The display objects may be changed in hue. Alternatively,the display objects may be changed in degree of focusing. As a matter ofcourse, display control processes other than those in theabove-mentioned examples may be executed, or processes of a plurality oftypes may be executed.

In that case, in accordance with the virtual-image positions of thedisplay objects, the display control unit 234 makes settings of detailsof the display control process to be executed on the display objects.For example, in a case where a process of controlling the sizes of thedisplay objects is executed, the display control unit 234 sets the sizesof the display objects based on the virtual-image positions of thedisplay objects. Then, the display control unit 234 controls thedisplay-image processing unit 232 so as to reflect (apply) the settingsto the display objects generated by the image generating unit 231. Inother words, the display-image processing unit 232 adjusts the sizes ofthe display objects generated by the image generating unit 231 to thesizes set by the display control unit 234.

In this way, when the predetermined display control process ofemphasizing the sense of depth is executed on the display objects, thesense of depth of the display objects can be matched with the sense ofperspective in the real space. With this, images to be viewed by theuser are displayed in a visually natural state in a display area of thedisplay unit 112. As a result, a sense of discomfort to the user can bereduced. Further, this process allows the user to more easily graspdifferences in position between the lines of the display objects. Withthis, operability of the transmissive HMD 100 can be further enhanced.

The processes in the above-mentioned units are executed under thecontrol by the system controller 211. In other words, as in the case ofthe image display apparatus described in the first embodiment, thesystem controller 211 allows the real space to be transparently viewed,and displays the plurality of display objects on the display unit 112that is configured to display three-dimensional images in a manner thatthe plurality of display objects are arranged in a plurality of linesthat are different from each other in position in the depth direction onthe near side with respect to the real space.

(Display Example of Display Objects)

FIG. 4 illustrates a display example of the display objects that aredisplayed as described above. A quadrangular frame 300 illustrated inFIG. 4 is an angular field within the display area of the display unit112 at the time of the imaging by the imaging unit 251. Thisquadrangular frame 300 may be or need not be displayed on the displayunit 112.

Physical objects 311 and 312 illustrated in FIG. 4 are physical objectsin the real space, which exist on a depth side with respect to thedisplay unit 112 as viewed from the user. As illustrated in FIG. 4, onthe display unit 112, the display objects that are images such as menuicons (display objects 321 to 327 and display objects 331 to 335) aredisplayed in a manner of being superimposed on a scene in the real spaceincluding these physical objects.

Different roles (functions) are assigned respectively to the displayobjects. For example, there may be displayed a display object having afunction of a GUI to be selected by the user so that a predeterminedprocess assigned in advance thereto, such as activation of anapplication, is executed. Alternatively, there may be displayed adisplay object that notifies reception of e-mails, or a display objecthaving a notification function to notify radio wave receiving conditionsof wireless communication.

Further, these display objects are three-dimensional images each formedof a right-eye image and a left-eye image that have binocular parallaxand a convergence angle therebetween. In other words, these displayobjects each have a virtual-image position in the depth direction(displayed in a manner of existing at predetermined positions in thedepth direction). In still other words, depending on settings of thebinocular parallax and the convergence angle, these display objects canbe displayed at desired virtual-image positions (displayed in a mannerof existing at desired positions in the depth direction as viewed fromthe user).

In the example of FIG. 4, these display objects (display objects 321 to327, and display objects 331 to 335) are displayed in a manner ofexisting on the near side with respect to the real space as viewed fromthe user. In other words, virtual-image positions of these displayobjects are set on the near side with respect to the real space asviewed from the user.

Further, the virtual-image positions (positions in the depth direction)of the display objects 321 to 327 are aligned with each other.Similarly, the virtual-image positions (positions in the depthdirection) of the display objects 331 to 335 are aligned with eachother. Thus, as illustrated in FIG. 4, the display objects 321 to 327are displayed in a manner of being arrayed in a single horizontal lineas viewed from the user. Similarly, the display objects 331 to 335 arealso displayed in a manner of being arrayed in a single horizontal lineas viewed from the user.

Meanwhile, the virtual-image positions of the display objects 321 to 327are set on the near side with respect to the virtual-image positions ofthe display objects 331 to 335 as viewed from the user. In other words,the virtual-image positions of the display objects 331 to 335 are set onthe depth side with respect to the virtual-image positions of thedisplay objects 321 to 327 as viewed from the user. In still otherwords, as illustrated in FIG. 4, a plurality of display objects aredisplayed in a manner of being arranged in a plurality of lines that aredifferent from each other in position in the depth direction as viewedfrom the user.

FIG. 5 illustrates an example of such a positional relationship betweenthe virtual-image positions. Specifically, as illustrated in the exampleof FIG. 5, as viewed from a user 350 of the transmissive HMD 100, thedisplay objects 331 to 335 are displayed in a single horizontal line ina layer 352 at a predetermined position on a near side with respect to areal space 353 in front of the user 350. The display objects 321 to 327are displayed in a single horizontal line in a layer 351 at apredetermined position on a near side with respect to the layer 352.

In this way, when a plurality of objects are displayed on the near sidewith respect to the real space in a manner of being arranged in aplurality of lines that are different from each other in position in thedepth direction, the user can view the plurality of display objects thatare arranged in order in each of the lines, and hence can more easilysearched for or select target objects. In this way, the operability ofthe transmissive HMD 100 can be enhanced.

Note that, the layers of the display objects are not particularlylimited in number, and the display objects may be arrayed in three ormore layers. Further, as long as virtual-image positions (positions inthe depth direction) of the display objects are aligned with each other,the display objects are not particularly limited in arrangement. Thedisplay objects may be arrayed not only in the horizontal direction, butalso in a vertical direction, an oblique direction, or other arbitrarydirections. Alternatively, the display objects may be arrayed in stillother arbitrary patterns such as a circular pattern, a radial pattern,and a matrix.

Further, in the example of FIG. 4, a marker is added to the physicalobject 311 in the real space, and a display object 341 as the marker isdisplayed. This display object 341 is added to the physical object 311in the real space, and hence, in the example of FIG. 5, is displayed ina manner of existing in the real space 353 (substantially at the sameposition as that of the physical object 311). In other words, thisdisplay object 341 is displayed at a position on the depth side asviewed from the user with respect to both the above-mentioned displayobjects 321 to 327 and the above-mentioned display objects 331 to 335.

The display objects 321 to 327, and the display objects 331 to 335 arelinked to the user 350 (transmissive HMD 100) (local coordinates). Thus,when global coordinates of the transmissive HMD 100 (display unit 112)vary (move), global coordinates of those display objects vary inaccordance therewith. Note that, relative coordinates (localcoordinates) of those display objects with respect to the transmissiveHMD 100 (display unit 112) remain unchanged (are displayed at the samepositions in the display unit 112 also after the movement).

Meanwhile, the display object 341 is linked to the physical object 311(global coordinates) in the real space 353. Thus, even when the globalcoordinates of the transmissive HMD 100 (display unit 112) vary (move),global coordinates of the display object 341 remain unchanged (aredisplayed near the physical object 311 as prior to the movement). Notethat, relative coordinates (local coordinates) of the display object 341with respect to the transmissive HMD 100 (display unit 112) vary, andhence the display object 341 after the movement is not necessarilydisplayed on the display unit 112 (as long as the physical object 311 isdisplayed, the display object 341 is displayed aside).

Note that, in the example of FIG. 4, the display objects 321 to 327 maybe displayed beyond an imaging range (quadrangular frame 300). Further,in the above description, seven display objects are displayed in theline on the near side as viewed from the user, and five display objectsare displayed in the line on the depth side as viewed from the user.However, the number of the display objects in each of the lines is notparticularly limited.

Further, the lines (layers) in which the display objects are arrangedmay be controlled in accordance with the roles (functions) representedby images of the display objects. For example, as in the case of FIG. 4,display objects of a group of icons (group of icons for instructingexecution of processes such as making a phone-call and checking e-mails)that can be controlled (selected) by the user are arranged in the lineon the near side as viewed from the user (layer 351). Display objects ofa group of interrupt icons relating to the user (group of icons fornotifying update of SNS or reception of new e-mails) are arranged in theline on the depth side as viewed from the user (layer 352). Displayobjects of navigation icons or markers linked to the physical objects inthe real space (such as physical object 311) may be arranged in the realspace 353.

In this way, when the display objects are sorted according to type andarrayed in corresponding lines in the depth direction, the user can moreeasily search for or select target objects. In this way, the operabilityof the transmissive HMD 100 can be enhanced.

(Flow of Display Control Process)

The system controller 211 of the transmissive HMD 100 executes thedisplay control process so that the display objects are displayed asdescribed above. Description is made of an example of a flow of thisdisplay control process with reference to the flowchart of FIG. 6.

When this display control process is started, in Step S101, the systemcontroller 211 determines whether or not to display a menu. For example,when the system controller 211 determines to display the menu inresponse, for example, to acceptance of an instruction, for example,from the user, or through satisfaction of a predetermined condition, thesystem controller 211 advances the flow to Step S102.

In Step S102, the system controller 211 controls the display controlunit 234 so that virtual-image positions of display objects in the menuare located on a near side with respect to a real space. Specifically,in order to set the virtual-image positions of the display objects,binocular parallax and a convergence angle of each of the displayobjects are set by the display control unit 234 under the control by thesystem controller 211.

In this context, description is made of a general computation method forthe convergence angle. For example, as illustrated in FIG. 7A, aconvergence angle α is formed when the user (transmissive HMD 100) isaway from a virtual-image position of a display object (also referred toas virtual-image distance) by a distance “a.” As illustrated in FIG. 7B,a convergence angle β is formed when the display object comes closerfrom the position at the virtual-image distance “a” to a position at avirtual-image distance b. As illustrated in FIG. 7C, a convergence angleΥ is formed when the display object moves away from the position at thevirtual-image distance “a” to a position at a virtual-image distance c.The left and right pupils are separated away from each other by adistance D.

When D is 61.5 mm and “a” is 4,000 mm, α is 53 minutes. In a case whereone pixel in the display unit 112 corresponds to 3 minutes, when animage display position is shifted inward from a predetermined positionin the horizontal direction by an amount corresponding to one pixel, βis 56 minutes, and b is 225 mm.

Meanwhile, when the image display position in the display unit 112 isshifted outward from a predetermined position in the horizontaldirection by the amount corresponding to one pixel, Υ is 50 minutes, andc is 228 mm. In this way, the convergence angle varies in accordancewith the image display position, and hence the virtual-image distance(in other words, virtual-image position) of the display object can bearbitrarily changed.

By using, for example, such a method, the display control unit 234 setsthe binocular parallax and the convergence angle of each of the displayobjects. As a matter of course, the method of setting the binocularparallax and the convergence angle of each of the display objects is notparticularly limited, and methods other than that in the above-mentionedexample may be employed.

In Step S103, the system controller 211 controls the display controlunit 234 so that settings are made on the predetermined display controlprocess of emphasizing the sense of depth of the display objects, whichis executed on the display objects in accordance with positions in thedepth direction. Note that, detailed description of this display controlprocess is made below.

In Step S104, the system controller 211 controls the image generatingunit 231 and the display-image processing unit 232 throughintermediation of the display control unit 234 so that display objects(three-dimensional images) are generated in accordance with the varioussettings made in Steps S102 and S103.

In Step S105, the system controller 211 controls the display drive unit233 and the display unit 112 through intermediation of the displaycontrol unit 234 so that the display objects generated in Step S104 aredisplayed on the display unit 112.

When the process of Step S105 is ended, the flow proceeds to Step S106.Further, in Step S101, when the system controller 211 determines not todisplay the menu, for example, in a case where the instruction, forexample, from the user has not yet been accepted, or a case where thepredetermined condition has not yet been satisfied, the flow proceeds toStep S106.

In Step S106, the system controller 211 determines whether or not todisplay a marker. For example, when the system controller 211 determinesto display the marker, for example, in a case where a physical object inthe real space, to which a marker is added, is located in the displayarea of the display unit 112, the system controller 211 advances theflow to Step S107.

In Step S107, the system controller 211 controls the display controlunit 234 so that, as in the process of Step S102, a virtual-imageposition of the display object as the marker is set in accordance withpositions in the real space of the physical object to which the markeris added. Specifically, in order to set a virtual-image position of thephysical object to which the marker is added, binocular parallax and aconvergence angle of the display object as the marker are set by thedisplay control unit 234 under the control by the system controller 211.

In Step S108, as in Step S103, the system controller 211 controls thedisplay control unit 234 so that settings are made on the predetermineddisplay control process of emphasizing the sense of depth of the displayobject, which is executed on the display object in accordance withpositions in the depth direction.

In Step S109, as in the case of Step S104, the system controller 211controls the image generating unit 231 and the display-image processingunit 232 through intermediation of the display control unit 234 so thatthe display object (three-dimensional image) is generated in accordancewith the various settings made in Steps S107 and S108.

In Step S110, the system controller 211 controls the display drive unit233 and the display unit 112 through intermediation of the displaycontrol unit 234 so that, as in Step S105, the display object generatedin Step S109 is displayed on the display unit 112.

When the process of Step S110 is ended, the flow proceeds to Step S111.Further, in Step S106, when the system controller 211 determines not todisplay the marker, for example, in a case where the physical object inthe real space, to which the marker is added, is located out of thedisplay area of the display unit 112, the flow proceeds to Step S111.

In Step S111, the system controller 211 determines whether or not to endthe display control process. When the system controller 211 determinesnot to end the display control process, the flow returns to Step S101,and the subsequent processes are repeated. Further, in Step S111, whenthe system controller 211 determines to end the display control process,the display control process is ended.

When the sensor unit 221 executes the display control process in thisway, the operability of the transmissive HMD 100 can be enhanced.Further, the processes of Steps S103 and S108 enable, for example, thesense of depth of the display objects to be matched with the sense ofperspective in the real space. With this, images to be viewed by theuser are displayed in a visually natural state in the display area ofthe display unit 112. As a result, the sense of discomfort to the usercan be reduced. Further, those processes allow the user to more easilygrasp differences in position between the lines of the display objects.With this, the operability of the transmissive HMD 100 can be furtherenhanced.

(Size)

Next, description is made of an example of the display control in StepsS103 and S108 in the display control process described above, which isexecuted on the display objects.

As an example of this display control process, the sizes of the displayobjects may be controlled in accordance with the virtual-image positionsof the display objects (specifically, in accordance with the positionsin the depth direction of the lines in which the display objects arearranged). More specifically, display objects at virtual-image positionson a nearer side as viewed from the user may be displayed on a largerscale. In other words, display objects at virtual-image positions on adeeper side may be displayed on a smaller scale.

FIG. 8 illustrates this example. In the case of the example of FIG. 8,display objects 324 and 333, and the display object 341 are displayed onthe display unit 112. As indicated by double-headed arrows 361A to 361C,and by double-headed arrows 362A to 362C, the display object 324 at avirtual-image position on the nearest side as viewed from the user isdisplayed on the largest scale, the display object 333 at avirtual-image position on the second nearest side as viewed from theuser is displayed on the second largest scale, and the display object341 at a virtual-image position on the deepest side as viewed from theuser is displayed on the smallest scale.

In order to perform such display control, for example, it is onlynecessary to cause the system controller 211 to control the displaycontrol unit 234 so that the sizes of the display objects are set inaccordance with the virtual-image positions thereof, and to control thedisplay-image processing unit 232 so that sizes of display objectsgenerated by the image generating unit 231 are set to be equal to thesizes set by the display control unit 234 (in other words, settings madeby the display control unit 234 are reflected (applied) to the displayobjects generated by the image generating unit 231).

With this, the sense of depth of the display objects can be emphasized,and the operability of the transmissive HMD 100 can be further enhanced.

Note that, the differences in size between the display objects may beexpressed in proportion to distances from the transmissive HMD 100 tothe virtual-image positions (virtual-image distances), or may be variedin accordance with the virtual-image distances. Specifically, the sizesof the display objects may be varied to be larger as the virtual-imagedistances become smaller (magnified at higher rate).

(Aerial Perspective)

Further, as another example of the display control process, at least oneof value, chroma, and contrast of the display objects may be controlledin accordance with the virtual-image positions of the display objects(specifically, in accordance with the positions in the depth directionof the lines in which the display objects are arranged). In other words,the sense of depth of the display objects may be emphasized by airtransparence expression.

FIG. 9 illustrates this example. In this case as illustrated in FIG. 9,the at least one of value, chroma, and contrast of the display objectsis controlled so that display objects at virtual-image positions on anearer side as viewed from the user are more clearly visible. In otherwords, the at least one of value, chroma, and contrast of the displayobjects is controlled so that display objects at virtual-image positionson a deeper side as viewed from the user are viewed in a more blurredstate.

In order to perform such display control, for example, it is onlynecessary to cause the system controller 211 to control the displaycontrol unit 234 so that the at least one of value, chroma, and contrastof the display objects is set in accordance with the virtual-imagepositions thereof, and to control the display-image processing unit 232so that the at least one of value, chroma, and contrast of displayobjects generated by the image generating unit 231 is set to be equal tothe value specified by the display control unit 234 (in other words,settings made by the display control unit 234 are reflected (applied) tothe display objects generated by the image generating unit 231).

With this, the sense of depth of the display objects can be emphasized,and the operability of the transmissive HMD 100 can be further enhanced.

Note that, the differences in value, chroma, and contrast between thedisplay objects from line to line may be expressed in proportion to thedistances from the transmissive HMD 100 to the virtual-image positions(virtual-image distances), or may be varied in accordance with thevirtual-image distances.

(Linear Perspective)

Further, as still another example of the display control process, aninterval between the display objects in each of the lines may becontrolled in accordance with the virtual-image positions of the displayobjects (specifically, in accordance with the positions in the depthdirection of the lines in which the display objects are arranged).

FIG. 10 illustrates this example. In FIG. 10, as indicated bydouble-headed arrows 371A and 371B, and by double-headed arrows 372A and372B, an interval between the display objects is set to be larger in aline on the nearer side as viewed from the user. In other words, aninterval between the display objects is set to be smaller in a line onthe deeper side as viewed from the user.

In order to perform such display control, for example, it is onlynecessary to cause the system controller 211 to control the displaycontrol unit 234 so that the interval between the display objects ineach of the lines is set in accordance with the virtual-image positionsthereof, and to control the display-image processing unit 232 so thatthe interval between the display objects in each of the lines, which aregenerated by the image generating unit 231, is set to be equal to thevalue specified by the display control unit 234 (in other words,settings made by the display control unit 234 are reflected (applied) tothe display objects generated by the image generating unit 231).

With this, the sense of depth of the display objects can be emphasized,and the operability of the transmissive HMD 100 can be further enhanced.

Note that, the difference in interval between the display objects fromline to line may be expressed in proportion to the distances from thetransmissive HMD 100 to the virtual-image positions (virtual-imagedistances), or may be varied in accordance with the virtual-imagedistances.

(Texture Gradient)

Further, as yet another example of the display control process,intervals between lines in the depth direction of the display objectsmay be controlled in accordance with the virtual-image positions of thedisplay objects (specifically, in accordance with the positions in thedepth direction of the lines in which the display objects are arranged).

FIG. 11 illustrates this example. In FIG. 11, the display objects 324and 333, and a display object 391 are displayed at positions in thestated order from the near side as viewed from the user. In this case,as indicated by double-headed arrows 381 and 382, an interval betweenthe lines (upper and lower directions of the display objects) is set tobe larger on the nearer side as viewed from the user. In other words, aninterval between the lines (upper and lower directions of the displayobjects) is set to be smaller on the deeper side as viewed from theuser.

In order to perform such display control, for example, it is onlynecessary to cause the system controller 211 to control the displaycontrol unit 234 so that the intervals between lines in the depthdirection of the display objects are set in accordance with thevirtual-image positions of the display objects in the lines, and tocontrol the display-image processing unit 232 so that intervals betweenlines of display objects generated by the image generating unit 231 areset to be equal to the values specified by the display control unit 234(in other words, settings made by the display control unit 234 arereflected (applied) to the display objects generated by the imagegenerating unit 231).

With this, the sense of depth of the display objects can be emphasized,and the operability of the transmissive HMD 100 can be further enhanced.

(Shading)

Further, as yet another example of the display control process, thedisplay objects may be shaded in accordance with the virtual-imagepositions of the display objects (specifically, in accordance with thepositions in the depth direction of the lines in which the displayobjects are arranged). With this, the sense of depth of the displayobjects can be emphasized, and the operability of the transmissive HMD100 can be further enhanced.

In order to perform such display control, for example, it is onlynecessary to cause the system controller 211 to control the displaycontrol unit 234 so that how the display objects are shaded is set inaccordance with the virtual-image positions thereof, and to control thedisplay-image processing unit 232 so that display objects generated bythe image generating unit 231 are shaded as set by the display controlunit 234 (in other words, settings made by the display control unit 234are reflected (applied) to the display objects generated by the imagegenerating unit 231).

(Superimposition)

Alternatively, display positions of the display objects may controlledso that at least parts of the display objects arranged in a line on thedepth side as viewed from the user are hidden by the display objectsarranged in a line on the near side as viewed from the user.

FIG. 12 illustrates this example. In FIG. 12, display positions of thedisplay object 324 and the display object 333 are controlled so that apart of the display object 333 arranged on the line on the depth side asviewed from the user (at a virtual-image position on the depth side asviewed from the user) is hidden by the display object 324 arranged inthe line on the near side as viewed from the user (at a virtual-imageposition on the near side as viewed from the user).

In order to perform such display control, for example, it is onlynecessary to cause the system controller 211 to control the displaycontrol unit 234 so that display positions of the display objects areset in accordance with the virtual-image positions thereof, wherebyparts of the display objects displayed near and far in the depthdirection are superimposed on each other, and to control thedisplay-image processing unit 232 so that display objects generated bythe image generating unit 231 are processed, whereby the superimpositionset by the display control unit 234 is expressed (in other words,settings made by the display control unit 234 are reflected (applied) tothe display objects generated by the image generating unit 231).

In this way, when the display objects arrayed in the depth direction aredisplayed in a superimposed manner, the sense of depth of the displayobjects is emphasized by a positional relationship therebetween. In thisway, the operability of the transmissive HMD 100 can be furtherenhanced.

(Hue)

Further, as yet another example of the display control process, hues ofthe display objects may be controlled in accordance with thevirtual-image positions of the display objects (specifically, inaccordance with the positions in the depth direction of the lines inwhich the display objects are arranged). Specifically, display objectsin warmer colors may be arranged in the line on the nearer side asviewed from the user, and display objects in cold colors may be arrangedin the line on the depth side as viewed from the user.

In order to perform such display control, for example, it is onlynecessary to cause the system controller 211 to control the displaycontrol unit 234 so that the hues of the display objects are set inaccordance with the virtual-image positions thereof, and to control thedisplay-image processing unit 232 so that hues of display objectsgenerated by the image generating unit 231 are set to the same hues setby the display control unit 234 (in other words, settings made by thedisplay control unit 234 are reflected (applied) to the display objectsgenerated by the image generating unit 231).

With this, the sense of depth of the display objects can be emphasizedby controlling the hues of the display objects, and the operability ofthe transmissive HMD 100 can be further enhanced.

(Focal Point)

Further, as yet another example of the display control process, degreesof focusing of the display objects may be controlled in accordance withthe virtual-image positions of the display objects (specifically, inaccordance with the positions in the depth direction of the lines inwhich the display objects are arranged). Specifically, display objectsarranged in the line on the nearest side as viewed from the user may befocused (degree of focusing becomes largest thereat), and displayobjects arranged in the line on the deeper side as viewed from the usermay be blurred in accordance therewith (defocused, in other words,degree of focusing is reduced thereat).

In order to perform such display control, for example, it is onlynecessary to cause the system controller 211 to control the displaycontrol unit 234 so that the degrees of focusing (image blur amounts,specifically, sharpnesses) of the display objects are set in accordancewith the virtual-image positions thereof, and to control thedisplay-image processing unit 232 so that degrees of focusing of displayobjects generated by the image generating unit 231 are set to be equalto the values specified by the display control unit 234 (in other words,settings made by the display control unit 234 are reflected (applied) tothe display objects generated by the image generating unit 231).

With this, the sense of depth of the display objects can be emphasizedby controlling blur amounts of the display objects (degrees ofdefocusing), and the operability of the transmissive HMD 100 can befurther enhanced.

Note that, the examples described above may be employed in variouscombinations. With this, the sense of depth of the display objects canbe further emphasized, and the operability of the transmissive HMD 100can be further enhanced.

(Movement)

Alternatively, the sensor unit 221 may include a detection sensorconfigured to detect changes in position or orientation of the displayunit 112. In a case where the detection sensor detects the variations inposition or orientation of the display unit 112, the display controlunit 234 may control, for example, the display-image processing unit 232or the display drive unit 233 so that the display positions of thedisplay objects on the display unit 112 are moved in accordance with thechanges.

Specifically, when the detection sensor starts to detect the changes inposition or orientation of the display unit 112, the display controlunit 234 moves the display positions of the display objects linked tolocal coordinates from positions with respect to which the display unit112 has not yet been changed in position or orientation into a directionopposite to a direction in which the display unit 112 has been changedin position or orientation. When the detection sensor detects that thechanges in position or orientation of the display unit 112 have ended,the display positions of the display objects may be returned to thepositions with respect to which the display unit 112 has not yet beenchanged in position or orientation.

(Horizontal Direction (Yaw Direction))

For example, display objects 323 to 325 and display objects 333 and 334each linked to local coordinates, and the display object 341 linked tothe global coordinates are displayed in the display area of the displayunit 112 as illustrated in FIG. 13A. When the user wearing thetransmissive HMD 100 as in the state illustrated in FIG. 13A moves thehead (face) in a horizontal direction (yaw direction) as illustrated inFIG. 13B, the display objects 323 to 325 and the display objects 333 and334 each linked to the local coordinates are moved as indicated byarrows 401A and 402A in FIG. 13B in a direction opposite to thedirection in which the transmissive HMD 100 is moved. In the case ofFIG. 13B, the user moves the head (face) to the right. Thus, the displayobjects 323 to 325 and the display objects 333 and 334 are moved to theleft in the display area as illustrated in FIG. 13B.

Then, as illustrated in FIG. 13C, when the user stops moving the head(face), the display objects 323 to 325 and the display objects 333 and334 are moved to the right in the display area as indicated by arrows401B and 402B. In this way, those display objects return to the samepositions as the display positions in the state illustrated in FIG. 13A.

With this, the user can feel as if the display objects displayed in thedisplay plane followed the movement of the user.

Further, as illustrated in FIG. 13B, the physical object 311 moves tothe left in the display area in conjunction with the movement of thehead (face) of the user. In conjunction therewith, the display object341 linked to the global coordinates moves to the left in the displayarea as indicated by an arrow 403. Note that, those movements are causedby a shift of a range of the real space within the display area of thedisplay unit 112. Thus, the physical object 311 and the display object341 as the marker thereof move independently of the movements of theabove-mentioned display objects linked to the local coordinates. Thus,as illustrated in FIG. 13C, when the user stops moving the head (face),the physical object 311 and the display object 341 stop moving inconjunction therewith, and hence do not return to their originalpositions unlike the other display objects linked to the localcoordinates.

In this way, the display objects linked to the local coordinates aremoved in conjunction with the movement of the display unit 112, andhence a sense of floating of the display objects linked to the localcoordinates can be emphasized with respect to the image of the realspace. Further, the display objects linked to the local coordinates aremoved differently from the display objects linked to the globalcoordinates and the physical object in the real space. With this, it canbe emphasized that the display objects linked to the local coordinatesare images relating to the transmissive HMD 100 (images indicatinginformation of the transmissive HMD 100).

Further, in that case, the display control unit 234 may be configured tomove the display objects arranged in the line on the nearer side asviewed from the user, for example, by an amount larger than an amount ofmoving the display objects arranged in the line on the deeper side asviewed from the user.

Specifically, in the case of FIGS. 13B and 13C, as indicated by thearrows 401A and 402A, and by the arrows 401B and402B, the displayobjects arranged in the line on the nearer side as viewed from the userare moved by a larger amount. With this, the sense of depth of thedisplay objects (differences in position in the depth direction of thedisplay objects in each of the lines) can be emphasized.

Still further, in that case, the display control unit 234 may beconfigured to move the display objects arranged in the line on thenearer side as viewed from the user, for example, earlier than thedisplay objects arranged in the line on the deeper side as viewed fromthe user.

Specifically, in the case of FIG. 13C, the display objects 323 to 325arranged in the line on the nearer side as viewed from the user aremoved at a timing earlier than that of the display objects 333 and 334.With this, the sense of depth of the display objects (differences inposition in the depth direction of the display objects in each of thelines) can be emphasized.

(Vertical Direction (Pitch Direction))

The same applies to a case of a movement in the vertical direction(pitch direction). For example, the display objects 323 to 325 and thedisplay objects 333 and 334 each linked to the local coordinates, andthe display object 341 linked to the global coordinates are displayed inthe display area of the display unit 112 as illustrated in FIG. 14A.When the user wearing the transmissive HMD 100 as in the stateillustrated in FIG. 14A moves the head (face) in a vertical direction(pitch direction) as illustrated in FIG. 14B, the display objects 323 to325 and the display objects 333 and 334 each linked to the localcoordinates are moved as indicated by arrows 411A and 412A in FIG. 14Bin a direction opposite to the direction in which the transmissive HMD100 is moved. In the case of FIG. 14B, the user moves the head (face)downward. Thus, the display objects 323 to 325 and the display objects333 and 334 are moved upward in the display area as illustrated in FIG.14B.

Then, as illustrated in FIG. 14C, when the user stops moving the head(face), the display objects 323 to 325 and the display objects 333 and334 are moved downward in the display area as indicated by arrows 411Band 412B. In this way, those display objects return to the samepositions as the display positions in the state illustrated in FIG. 14A.

With this, the user can feel as if the display objects displayed in thedisplay plane followed the movement of the user.

Further, as illustrated in FIG. 14B, the physical object 311 movesupward in conjunction with the movement of the head (face) of the user.In conjunction therewith, the display object 341 linked to the globalcoordinates moves to the right as indicated by an arrow 413. Note that,those movements are caused by a shift of a range of the real spacewithin the display area of the display unit 112. Thus, the physicalobject 311 and the display object 341 as the marker thereof moveindependently of the movements of the above-mentioned display objectslinked to the local coordinates. Thus, as illustrated in FIG. 14C, whenthe user stops moving the head (face), the physical object 311 and thedisplay object 341 stop moving in conjunction therewith, and hence donot return to their original positions unlike the other display objectslinked to the local coordinates.

In this way, the display objects linked to the local coordinates aremoved in conjunction with the movement of the display unit 112, andhence a sense of floating of the display objects linked to the localcoordinates can be emphasized with respect to the image of the realspace. Further, the display objects linked to the local coordinates aremoved differently from the display objects linked to the globalcoordinates and the physical object in the real space. With this, it canbe emphasized that the display objects linked to the local coordinatesare images relating to the transmissive HMD 100 (images indicatinginformation of the transmissive HMD 100).

Further, in that case, the display control unit 234 may be configured tomove the display objects arranged in the line on the nearer side asviewed from the user, for example, by an amount larger than an amount ofmoving the display objects arranged in the line on the deeper side asviewed from the user.

Specifically, in the case of FIGS. 14B and 14C, as indicated by thearrows 411A and 412A, and by the arrows 411B and412B, the displayobjects arranged in the line on the nearer side as viewed from the userare moved by a larger amount. With this, the sense of depth of thedisplay objects (differences in position in the depth direction of thedisplay objects in each of the lines) can be emphasized.

Still further, in that case, the display control unit 234 may beconfigured to move the display objects arranged in the line on thenearer side as viewed from the user, for example, earlier than thedisplay objects arranged in the line on the deeper side as viewed fromthe user.

Specifically, in the case of FIG. 14C, the display objects 323 to 325arranged in the line on the nearer side as viewed from the user aremoved at a timing earlier than that of the display objects 333 and 334.With this, the sense of depth of the display objects (differences inposition in the depth direction of the display objects in each of thelines) can be emphasized.

The processes described above of each unit are executed under thecontrol by the system controller 211.

(Display Control Process)

FIG. 15A illustrates an example of how to specify a moving direction andcalculate a moving amount based on detected movement of the transmissiveHMD 100 according to this embodiment. Further, FIG. 15B illustrates anexample of a reference position and an example of coordinate axesdefined in the display plane according to this embodiment. In theexample illustrated in FIG. 15B, the X-axis corresponding to thehorizontal direction and the Y-axis corresponding to the verticaldirection are defined in the display plane. Note that, the referenceposition according to this embodiment is not necessarily limited to theone point as illustrated in FIG. 15B, and may be set, for example, as aregion including a plurality of coordinate pairs.

Specifically, the transmissive HMD 100 forms an angle θ to be specifiedbased on detected data acquired from the sensor provided in the sensorunit 221, which is capable of detecting movement. Based, for example, onthe detected movement, specifically, on an angle θ₀ at a first timepoint and an angle θ₁ at a second time point (time point after the firsttime point) illustrated in FIG. 15A, the system controller 211 specifiesthe moving direction and calculates the moving amount by the followingequation (1) below. In the equation (1), the plus sign and the minussign of Δθ each indicate the moving direction, and an absolute value ofΔθ represents the moving amount. Note that, in the examples illustratedin FIG. 15, a turning direction indicated by an arrow in FIG. 15A isdefined as a positive direction.Δθ=θ₁−θ₀  (1)

Next, with reference to the flowchart of FIG. 16, description is made ofan example of a flow of another display control process in a case wherea display object is moved as described above using such parameters.

When this display control process is started, in Step S201, the systemcontroller 211 determines whether or not the calculated absolute valueof Δθ (in other words, movement amount according to this embodiment) isequal to or more than a preset threshold. This preset threshold may be apreset fixed value, or may be a variable that can be set as appropriate,for example, through operation by the user.

Note that, for example, in Step S201, the system controller 211 maydetermine whether or not the calculated absolute value of Δθ is largerthan the preset threshold.

When the system controller 211 determines that the absolute value of Δθis equal to or more than the preset threshold, the flow proceeds to StepS202. In Step S202, the system controller 211 determines whether or notthe calculated Δθ is negative.

When the system controller 211 determines that the calculated Δθ isnegative, the flow proceeds to Step S203. In Step S203, the systemcontroller 211 controls the display-image processing unit 232 or thedisplay drive unit 233 via the display control unit 234 so that thedisplay object is moved in a positive direction (direction of X>0) onthe X-axis in the display plane. When the process of Step S203 is ended,the display control process is ended.

Further, in Step S202, when the system controller 211 determines thatthe calculated Δθ is positive, the flow proceeds to Step S204. In StepS204, the system controller 211 controls the display-image processingunit 232 or the display drive unit 233 via the display control unit 234so that the display object is moved in a negative direction (directionof X<0) on the X-axis in the display plane. When the process of StepS204 is ended, the display control process is ended.

Further, in Step S201, when the system controller 211 determines thatthe absolute value of Δθ is not equal to or more than the presetthreshold, the flow proceeds to Step S205. In Step S205, the systemcontroller 211 controls the display-image processing unit 232 or thedisplay drive unit 233 via the display control unit 234 so that whetheror not the display object has come to the reference position isdetermined. When the system controller 211 determines that currentcoordinates of the display object and coordinates of the referenceposition are the same as each other, the system controller 211determines that the display object has come to the reference position,and the display control process is ended.

Further, in Step S205, when the system controller 211 determines thatthe display object has not come to the reference position, the flowproceeds to Step S206. In Step S206, the system controller 211 controlsthe display-image processing unit 232 or the display drive unit 233 viathe display control unit 234 so that the display object is moved towardthe reference position. When the process of Step S206 is ended, the flowreturns to Step S201, and the subsequent steps are repeated.

By executing the display control process in this way, the user can feelas if the display object displayed in the display plane followed themovement of the user.

Note that, as described above, in order to move the display objectsarranged in the line on the nearer side by an amount larger than anamount of moving the display objects arranged in the line on the deeperside as viewed from the user, it is only necessary to cause the systemcontroller 211 (display control unit 234) to control corresponding unitsso that those display objects are moved differently from display objectsin other lines, that is, moved in such a manner (larger amount in theline on the nearer side) by the processes of Steps S203, S204, and S206.

Further, as described above, in order to move the display objectsarranged in the line on the nearer side earlier than the display objectsarranged in the line on the deeper side as viewed from the user, it isonly necessary to cause the system controller 211 (display control unit234) to control corresponding units so that those display objects aremoved differently from display objects in other lines, that is, moved insuch a manner (earlier in the lines on the nearer side) by the processesof Step S203, S204, and S206.

(Turning Direction (Roll Direction))

Note that, as for movements in a turning direction (roll direction), notonly the display objects linked to the local coordinates but also thedisplay objects linked to the global coordinates roll in a directionopposite to a direction in which the user tilts the head (face) and atan angle at which the user tilts the head (face). In other words, howthe display objects are displayed is controlled in a manner that thehorizontal direction (or vertical direction) in the real space and thehorizontal direction (or vertical direction) of the display objects arealways matched with each other.

Specifically, when the user wearing the transmissive HMD 100 as in thestate illustrated in FIG. 17A moves the head (face) to the left in theturning direction (roll direction) as illustrated in FIG. 17B, thedisplay objects 323 to 325 and the display objects 333 and 334 that aredisplayed in the display area of the display unit 112 as in FIG. 17Aturn to the right in the turning direction in the display area at anangle equal to a turning angle of the transmissive HMD 100 asillustrated in FIG. 17B.

Further, specifically, when the user wearing the transmissive HMD 100 asin the state illustrated in FIG. 17A moves the head (face) to the rightin the turning direction (roll direction) as illustrated in FIG. 17C,the display objects 323 to 325 and the display objects 333 and 334 thatare displayed in the display area of the display unit 112 as in FIG. 17Aturn to the left in the turning direction in the display area at anangle equal to a turning angle of the transmissive HMD 100 asillustrated in FIG. 17C.

Note that, in this case, as illustrated in FIGS. 17B and 17C, thephysical object 311 and the display object 341 linked to the globalcoordinates turn as well as the display objects linked to the localcoordinates.

(Display Control Process)

In the following, turning directions indicated by the arrows in FIGS.18A and 18B are each defined as a positive direction. With reference tothe flowchart of FIG. 19, description is made of an example of a flow ofstill another display control process in a case where a display objectis turned as described above.

When this display control process is started, in Step S301, the systemcontroller 211 calculates a tilt θ of the display apparatus(transmissive HMD 100). The system controller 211 calculates the tilt θbased, for example, on detected data acquired from the sensor providedin the sensor unit 221, which is capable of detecting the movement ofthe transmissive HMD 100. Note that, the plus sign and the minus sign ofthe tilt θ each indicates a tilt direction, and an absolute value of thetilt θ represents a tilt amount.

When the tilt θ (in other words, tilt direction and tilt amount inaccordance with the detected movement) is calculated, in Step S302, thesystem controller 211 controls the display-image processing unit 232 orthe display drive unit 233 via the display control unit 234 so that thedisplay object is turned by an amount corresponding to the absolutevalue of the tilt θ (in other words, tilt amount) in a directionopposite to the direction indicated by the sign of the tilt θ (in otherwords, tilt direction). In this way, the display object is moved.

When the process of Step S302 is ended, the display control process isended. By executing the display control process in this way,irrespective of the tilt of the display unit 112, the horizontaldirection (vertical direction) of the display object displayed on thedisplay unit 112 can be always matched with the horizontal direction(vertical direction) in the real space on which the display object issuperimposed. Thus, images are visually naturally visible to the user,and hence the sense of discomfort to the user can be reduced. With this,the operability of the transmissive HMD 100 can be further enhanced.

In the description hereinabove, the display unit 112 is a transmissivedisplay configured to transmit light. However, a non-transmissivedisplay configured not to transmit light may be employed as the displayunit 112. Specifically, on the display unit 112 (non-transmissivedisplay), images to be viewed from a plurality of viewpoints (forexample, display objects such as a menu icon) may be displayed as athree-dimensional image in a superimposed manner on captured imagesobtained by the imaging unit 251 (also referred to as through images).Also in that case, as in the case of the transmissive display describedhereinabove, it is only necessary to display the display objects on thedisplay unit 112 in a manner that the display objects are arranged in aplurality of lines that are different from each other in position in thedepth direction on a near side with respect to a scene in the realspace. In other words, the present technology is applicable, forexample, not only to the transmissive HMD 100 but also to anon-transmissive HMD.

The series of processes described hereinabove may be executed byhardware, or may be executed by software. In a case where the series ofprocesses described hereinabove is executed by software, programs of thesoftware are installed via a network or from recording media.

Examples of the recording media include the removable medium 264 that isdelivered separately from the apparatus main body as illustrated, forexample, in FIG. 3 so that programs stored therein are distributed tothe user. Examples of the removable media 264 include not only magneticdisks (such as flexible disk) and optical disks (such as a CD-ROM and aDVD), but also magneto-optical disks (such as an MD (MiniDisc)) and asemiconductor memory.

In that case, such removable media are mounted to the drive so that theprograms can be installed to the storage unit 261.

Alternatively, those programs may be provided via wired or wirelesstransmission media such as a local area network, the Internet, anddigital satellite broadcasting. In those cases, the programs may bereceived via the communication unit 262, and then installed to thestorage unit 261 of the apparatus.

Still alternatively, those programs may be pre-installed in the ROM ofthe system controller 211 or the storage unit 261.

Note that, the programs to be executed by the computer may be executedin time series in the order described in this specification, or may beexecuted parallel to each other or at necessary timings such as a timingof a call.

Further, in this specification, the steps describing the programs storedin the recording media include, as a matter of course, not onlyprocesses to be executed in time series in the order describedhereinabove, but also processes to be executed not necessarily in timeseries, in other words, executed in parallel or individually.

Still further, the processes of the steps described hereinabove may beexecuted in the apparatus described hereinabove, or may be executed inany other apparatus than the apparatus described hereinabove. In thatcase, it is only necessary that the apparatus to execute the processeshave functions that are necessary for executing the processes (such asfunction blocks). Further, it is only necessary to transmit informationitems that are necessary for the processes as appropriate to theapparatus.

In addition, in this specification, the “system” refers to a collectionof a plurality of components (such as apparatus and modules (parts)),and all the components need not necessarily be provided in the samecasing. Thus, both a plurality of apparatus that are containedrespectively in their casings and connected to each other via a network,and a single apparatus that has a single casing containing a pluralityof modules are encompassed in the definition of the “system.”

Further, the configuration described as a single apparatus (orprocessing unit) hereinabove may be divided into a plurality ofapparatus (or processing units). In contrast, the configurationsdescribed as a plurality of apparatus (or processing units) hereinabovemay be integrated into a single apparatus (or processing unit). Stillfurther, as a matter of course, configurations other than thosedescribed hereinabove may be added to the configurations of theapparatus (or processing units). Yet further, as long as theconfigurations and operations of the entire system are substantiallyunchanged, a part of a configuration of a certain apparatus (orprocessing unit) may be incorporated in a configuration of anotherapparatus (or another processing unit).

The technical scope of the present disclosure, which is described indetail hereinabove in the preferred embodiments of the presentdisclosure with reference to the accompanying drawings, is not limitedto those examples. It is obvious that various changes and modificationscould have been made by those who have common knowledge in the technicalfield of the present disclosure within the technical scope described in“What is claimed is.” It should be understood that those changes andmodifications obviously belong to the technical scope of the presentdisclosure.

For example, the present technology may include a system of cloudcomputing in which a single function is shared with and cooperativelyexerted in a plurality of apparatus via a network.

Further, the steps described above with reference to the flowcharts maybe executed in a single apparatus, or may be shared with and executed ina plurality of apparatus.

Still further, in a case where a plurality of processes are contained ina single step, the plurality of processes contained in the single stepmay be executed in a single apparatus, or may be shared with andexecuted in a plurality of apparatus.

In addition, the present technology is not limited thereto, and may becarried out by any type of configuration to be mounted to such apparatusor to apparatus having such systems, specifically, a processor as, forexample, system LSI (Large Scale Integration), a module using aplurality of the processors and the like, a unit using a plurality ofthe modules, and a set obtained by adding other functions to the unit(that is, a part of a configuration of an apparatus).

Note that, the present technology may employ the followingconfigurations.

(1) An image display apparatus, including:

a display unit configured to allow a real space to be transparentlyviewed and configured to display a three-dimensional image; and

a display control unit configured to display a plurality of thethree-dimensional images on the display unit in a manner that theplurality of the three-dimensional images are arranged in a plurality oflines that are different from each other in position in a depthdirection on a near side with respect to the real space.

(2) The image display apparatus according to any one of Item (1) andItems (3) to (13), in which the display control unit controls sizes ofthe plurality of the three-dimensional images in accordance with thepositions in the depth direction of the plurality of lines in which theplurality of the three-dimensional images are arranged.

(3) The image display apparatus according to any one of Items (1) and(2) and Items (4) to (13), in which the display control unit controls atleast one of value, chroma, and contrast of the plurality of thethree-dimensional images in accordance with the positions in the depthdirection of the plurality of lines in which the plurality of thethree-dimensional images are arranged.

(4) The image display apparatus according to any one of Items (1) to (3)and Items (5) to (13), in which the display control unit controls, inaccordance with the positions in the depth direction of the plurality oflines in which the plurality of the three-dimensional images arearranged, an interval between the plurality of the three-dimensionalimages in each of the plurality of lines.

(5) The image display apparatus according to any one of Items (1) to (4)and Items (6) to (13), in which the display control unit controls, inaccordance with the positions in the depth direction of the plurality oflines in which the plurality of the three-dimensional images arearranged, an interval between the plurality of lines in the depthdirection.

(6) The image display apparatus according to any one of Items (1) to (5)and Items (7) to (13),

in which the plurality of lines in which the plurality of thethree-dimensional images are arranged include

-   -   a line on a depth side as viewed from a user, and    -   a line on the near side as viewed from the user,

in which the plurality of the three-dimensional images include

-   -   a plurality of three-dimensional images that are arranged in the        line on the depth side as viewed from the user, and    -   a plurality of three-dimensional images that are arranged in the        line on the near side as viewed from the user, and

in which the display control unit controls display positions of theplurality of the three-dimensional images so that at least parts of theplurality of three-dimensional images that are arranged in the line onthe depth side as viewed from the user are hidden by the plurality ofthree-dimensional images that are arranged in the line on the near sideas viewed from the user.

(7) The image display apparatus according to any one of Items (1) to (6)and Items (8) to (13), in which the display control unit controls huesof the plurality of the three-dimensional images in accordance with thepositions in the depth direction of the plurality of lines in which theplurality of the three-dimensional images are arranged.

(8) The image display apparatus according to any one of Items (1) to (7)and Items (9) to (13), in which the display control unit controlsdegrees of focusing of the plurality of the three-dimensional images inaccordance with the positions in the depth direction of the plurality oflines in which the plurality of the three-dimensional images arearranged.

(9) The image display apparatus according to any one of Items (1) to (8)and Items (10) to (13), in which the display control unit controls, inaccordance with roles assigned respectively to the plurality of thethree-dimensional images, in which of the plurality of lines theplurality of the three-dimensional images are arranged.

(10) The image display apparatus according to any one of Items (1) to(9) and Items (11) to (13), further including a detection unitconfigured to detect changes in at least one of position and orientationof the display unit,

in which, when the detection unit detects the changes in the at leastone of position and orientation of the display unit, the display controlunit causes the plurality of the three-dimensional images to be moved inaccordance with the changes.

(11) The image display apparatus according to any one of Items (1) to(10) and Items (12) and (13),

in which, when the detection unit starts to detect the changes in the atleast one of position and orientation of the display unit, the displaycontrol unit causes display positions of the plurality of thethree-dimensional images to be moved from positions with respect towhich the display unit has not yet been changed in any of position andorientation into a direction opposite to a direction in which thedisplay unit has been changed in the at least one of position andorientation, and

in which, when the detection unit detects that the changes in the atleast one of position and orientation have ended, the display controlunit causes the display positions of the plurality of thethree-dimensional images to be returned to the positions with respect towhich the display unit has not yet been changed in the any of positionand orientation.

(12) The image display apparatus according to any one of Item (1) andItems (11) to (13),

in which the plurality of lines in which the plurality of thethree-dimensional images are arranged include

-   -   a line on a deeper side as viewed from a user, and    -   a line on a nearer side as viewed from the user,

in which the plurality of the three-dimensional images include

-   -   a plurality of three-dimensional images that are arranged in the        line on the deeper side as viewed from the user, and    -   a plurality of three-dimensional images that are arranged in the        line on the nearer side as viewed from the user, and

in which the display control unit causes the plurality ofthree-dimensional images that are arranged in the line on the nearerside as viewed from the user to be moved by an amount larger than anamount of moving the plurality of three-dimensional images that arearranged in the line on the deeper side as viewed from the user.

(13) The image display apparatus according to any one of Items (1) to(12),

in which the plurality of lines in which the plurality of thethree-dimensional images are arranged include

-   -   a line on a deeper side as viewed from a user, and    -   a line on a nearer side as viewed from the user,

in which the plurality of the three-dimensional images include

-   -   a plurality of three-dimensional images that are arranged in the        line on the deeper side as viewed from the user, and    -   a plurality of three-dimensional images that are arranged in the        line on the nearer side as viewed from the user, and

in which the display control unit causes the plurality ofthree-dimensional images that are arranged in the line on the nearerside as viewed from the user to be moved earlier than the plurality ofthree-dimensional images that are arranged in the line on the deeperside as viewed from the user.

(14) An image display method, including displaying a plurality ofthree-dimensional images on a display unit in a manner that theplurality of three-dimensional images are arranged in a plurality oflines that are different from each other in position in a depthdirection on a near side with respect to the real space, the displayunit being configured to allow a real space to be transparently viewedand configured to display the plurality of three-dimensional images.

What is claimed is:
 1. An image display apparatus, comprising: a displayunit configured to allow a real space to be transparently viewed andconfigured to display a plurality of three-dimensional images; adetection unit configured to detect changes in at least one of positionand orientation of the display unit; and a display control unitconfigured to display the plurality of three-dimensional images on thedisplay unit in a manner that multiple images of the plurality ofthree-dimensional images are arranged in each line of a plurality oflines that are different from each other in position in a depthdirection on a near side with respect to the real space, wherein, whenthe detection unit detects the changes in the at least one of positionand orientation of the display unit, the display control unit causes theplurality of the three-dimensional images to be moved in accordance withthe changes, and wherein the display unit and the display control unitare each implemented via at least one processor.
 2. The image displayapparatus according to claim 1, wherein the display control unitcontrols sizes of the plurality of the three-dimensional images inaccordance with the positions in the depth direction of the plurality oflines in which the plurality of the three-dimensional images arearranged.
 3. The image display apparatus according to claim 1, whereinthe display control unit controls at least one of value, chroma, andcontrast of the plurality of the three-dimensional images in accordancewith the positions in the depth direction of the plurality of lines inwhich the plurality of the three-dimensional images are arranged.
 4. Theimage display apparatus according to claim 1, wherein the displaycontrol unit controls, in accordance with the positions in the depthdirection of the plurality of lines in which the plurality of thethree-dimensional images are arranged, an interval between the pluralityof the three-dimensional images in each of the plurality of lines. 5.The image display apparatus according to claim 1, wherein the displaycontrol unit controls, in accordance with the positions in the depthdirection of the plurality of lines in which the plurality of thethree-dimensional images are arranged, an interval between the pluralityof lines in the depth direction.
 6. The image display apparatusaccording to claim 1, wherein the plurality of lines in which theplurality of the three-dimensional images are arranged include a line ona depth side as viewed from a user, and a line on the near side asviewed from the user, wherein the plurality of the three-dimensionalimages include a plurality of three-dimensional images that are arrangedin the line on the depth side as viewed from the user, and a pluralityof three-dimensional images that are arranged in the line on the nearside as viewed from the user, and wherein the display control unitcontrols display positions of the plurality of the three-dimensionalimages so that at least parts of the plurality of three-dimensionalimages that are arranged in the line on the depth side as viewed fromthe user are hidden by the plurality of three-dimensional images thatare arranged in the line on the near side as viewed from the user. 7.The image display apparatus according to claim 1, wherein the displaycontrol unit controls hues of the plurality of the three-dimensionalimages in accordance with the positions in the depth direction of theplurality of lines in which the plurality of the three-dimensionalimages are arranged.
 8. The image display apparatus according to claim1, wherein the display control unit controls degrees of focusing of theplurality of the three-dimensional images in accordance with thepositions in the depth direction of the plurality of lines in which theplurality of the three-dimensional images are arranged.
 9. The imagedisplay apparatus according to claim 1, wherein the display control unitcontrols, in accordance with roles assigned respectively to theplurality of the three-dimensional images, the plurality of lines inwhich the plurality of the three-dimensional images are arranged. 10.The image display apparatus according to claim 1, wherein, when thedetection unit starts to detect the changes in the at least one ofposition and orientation of the display unit, the display control unitcauses display positions of the plurality of the three-dimensionalimages to be moved from positions with respect to which the display unithas not yet been changed in any of position and orientation into adirection opposite to a direction in which the display unit has beenchanged in the at least one of position and orientation, and wherein,when the detection unit detects that the changes in the at least one ofposition and orientation have ended, the display control unit causes thedisplay positions of the plurality of the three-dimensional images to bereturned to the positions with respect to which the display unit has notyet been changed in the any of position and orientation.
 11. The imagedisplay apparatus according to claim 10, wherein the plurality of linesin which the plurality of the three-dimensional images are arrangedinclude a line on a deeper side as viewed from a user, and a line on anearer side as viewed from the user, wherein the plurality of thethree-dimensional images include a plurality of three-dimensional imagesthat are arranged in the line on the deeper side as viewed from theuser, and a plurality of three-dimensional images that are arranged inthe line on the nearer side as viewed from the user, and wherein thedisplay control unit causes the plurality of three-dimensional imagesthat are arranged in the line on the nearer side as viewed from the userto be moved by an amount larger than an amount of moving the pluralityof three-dimensional images that are arranged in the line on the deeperside as viewed from the user.
 12. The image display apparatus accordingto claim 10, wherein the plurality of lines in which the plurality ofthe three-dimensional images are arranged include a line on a deeperside as viewed from a user, and a line on a nearer side as viewed fromthe user, wherein the plurality of the three-dimensional images includea plurality of three-dimensional images that are arranged in the line onthe deeper side as viewed from the user, and a plurality ofthree-dimensional images that are arranged in the line on the nearerside as viewed from the user, and wherein the display control unitcauses the plurality of three-dimensional images that are arranged inthe line on the nearer side as viewed from the user to be moved earlierthan the plurality of three-dimensional images that are arranged in theline on the deeper side as viewed from the user.
 13. An image displaymethod, performed via at least one processor, the method comprising:detecting changes in at least one of position and orientation of adisplay unit; and displaying a plurality of three-dimensional images onthe display unit in a manner that multiple images of the plurality ofthree-dimensional images are arranged in each line of a plurality oflines that are different from each other in position in a depthdirection on a near side with respect to the real space, the displayunit being configured to allow a real space to be transparently viewedand configured to display the plurality of three-dimensional images,wherein the detected changes in the at least one of position andorientation of the display unit cause the plurality of thethree-dimensional images to be moved in accordance with the changes.